With a sufficiently large
hyper parameter search space, it could take many iterations or child
runs to try every possible combination. Typically, you set a maximum
number of iterations but this could still result in a large
number of runs that don't result in a better model than a combination
that has already been tried. To help prevent wasting time you can
set an early termination policy that abandons runs. The runs that are abandoned
are unlikely to produce a better result than the previously completed runs. This policy is evaluated at an evaluation
interval you specify based on each time the target performance metric is logged. You can also set a delay evaluation
parameter in order to avoid evaluating the policy until the minimum number
of iterations have been completed. It is important to note early
termination is particularly useful for deep learning scenarios where
a deep neural network or DNN is trained iterative lee
over a number of epochs. The training script can report
the target metric after each epoch. And if the run is significantly
underperforming previous runs after the same number of intervals,
it can be abandoned. You can use a bandit policy to stop
a run if the target performance metric underperforms the best run so
far by a specified margin. This example applies the policy for
every iteration After the 1st 5. and abandons runs where the reported
target metric is 0.2 or more worse than the best performing run
after the same number of intervals. You can also apply a bandit
policy using a slack factor, which compares the performance metric as
a ratio rather than an absolute value. A median stopping policy abandons runs
with the target performance metric is worse than the medium of
the running averages for all runs. A truncation selection policy cancels
the lowest performing X percent of runs at each evaluation interval, based on the truncation percentage
value you specify for X.